User-configurable interactive region monitoring

ABSTRACT

In one embodiment, one or more systems may receive input from a user identifying an interactive region of a physical environment. The one or more systems may determine a location of the interactive region relative to a depth sensor and monitor, at least in part by the depth sensor, the interactive region for a predetermined event. The one or more systems may detect, at least in part by the depth sensor, the predetermined event. In response to detecting the predetermined event, the one or more systems may initiate a predetermined action associated with the predetermined event.

PRIORITY

This application claims the benefit as a continuation under 35 U.S.C. §120 of U.S. patent application Ser. No. 15/015,040 titled“User-Configurable Interactive Region Monitoring” and filed on 3 Feb.2016, which claims the benefit, under 35 U.S.C. § 119(e), of U.S.Provisional Patent Application No. 62/208,265 filed 21 Aug. 2015, bothof which are incorporated herein by reference.

TECHNICAL FIELD

This disclosure generally relates to electronic event detection.

BACKGROUND

A depth sensor may generate a depth image that includes an array ofpixels corresponding to regions of a physical environment with a depthvalue for each pixel indicating the depth of the corresponding region.Depth images may be generated by stereo triangulation, sheet of lighttriangulation, structured light, interferometry, coded aperture, orother means.

Connected devices may be part of a communication network such as a localarea network, wide area network, cellular network, the Internet, or anyother suitable network. A connected device may use a communicationnetwork to communicate with other devices, for example, to accessremotely-stored data, access remote processing power, access remotedisplays, provide locally-stored data, provide local processing power,or provide access to local displays. For example, networks may providecommunication paths and links to servers, which may host applications,content, and services that may be accessed or utilized by connecteddevices. The content may include text, video data, audio data, usersettings or other types of data. Networks may use any suitablecommunication protocol or technology to facilitate communication betweenconnected devices, such as, for example, BLUETOOTH, IEEE WI-FI(802.11a/b/g/n/ac), or TCP/IP.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example network environment associated with aninteractive region monitoring system.

FIG. 2 illustrates an example interactive region in an example physicalenvironment.

FIGS. 3A-D illustrate example embodiments of a user configuring aninteractive region.

FIG. 4 illustrates multiple example interactive regions in an examplephysical environment.

FIG. 5 illustrates an example user interface for configuring interactiveregions, predetermined events, event conditions, and predeterminedactions.

FIG. 6 illustrates an example method implemented by an exampleinteractive monitoring system.

FIG. 7 illustrates an example method for detecting a predetermined eventusing an image capture by a depth sensor.

FIG. 8 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Particular embodiments allow a user to configure interactive regions ina physical environment. The interactive regions may be monitored by aninteractive region monitoring system for one or more predeterminedevents, such as, for example, events that have been defined by a user,automatically defined by a computing device, or both. In particularembodiments, the interactive region monitoring system may include one ormore depth sensors. When one or more predetermined events are detected,a predetermined action may be initiated by the interactive regionmonitoring system. In some embodiments, the interactive regionmonitoring system communicates electronically with another connecteddevice to initiate the action. An interactive region monitoring systemmay include one or more depth sensors, connected devices, servers,client systems, or any other devices or systems. Although thisdisclosure describes and illustrates particular interactive regionmonitoring systems, this disclosure contemplates any suitableinteractive region monitoring system for creating interactive regions.In particular embodiments, an interactive region monitoring system maybe part of, or connected to, a depth-sensing computing device.

FIG. 1 illustrates an example network environment 100 for particularembodiments of an interactive region monitoring system. Networkenvironment 100 may include connected devices 130, data store 140,interactive region monitoring server 150, and sensors 160 connected toeach other by network 110. Although FIG. 1 illustrates a particulararrangement of connected devices 130, data store 140, interactive regionmonitoring server 150, and sensors 160, this disclosure contemplates anysuitable arrangement of connected devices 130, data store 140,interactive region monitoring server 150, and sensors 160. As an exampleand not by way of limitation, two or more of connected devices 130, datastore 140, interactive region monitoring server 150, and sensors 160 maybe connected to each other directly, bypassing network 110. As anotherexample, two or more of connected devices 130, data store 140,interactive region monitoring server 150, and sensors 160 may bephysically or logically co-located with each other in whole or in part.As an example, data store 140 may be directly connected to or co-locatedwith interactive region monitoring server 150. As another example, oneor more sensors 160 may be directly connected to or co-located with oneor more connected devices 130. Moreover, although FIG. 1 illustrates aparticular number of connected devices 130, data stores 140, interactiveregion monitoring servers 150, sensors 160, and networks 110 thisdisclosure contemplates any suitable number of connected devices 130,data stores 140, interactive region monitoring servers 150, sensors 160,and networks 110. As an example and not by way of limitation, networkenvironment 100 may include multiple connected devices 130, data stores140, interactive region monitoring servers 150, sensors 160, andnetworks 110.

This disclosure contemplates any suitable network 110. As an example andnot by way of limitation, one or more portions of network 110 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 110 may include one or more networks110.

Links 120 may connect connected devices 130, data store 140, interactiveregion monitoring server 150, and sensors 160 to communication network110 or to each other. This disclosure contemplates any suitable links120. In particular embodiments, one or more links 120 include one ormore wireline (such as for example Digital Subscriber Line (DSL) or DataOver Cable Service Interface Specification (DOCSIS)), wireless (such asfor example Wi-Fi or Worldwide Interoperability for Microwave Access(WiMAX)), or optical (such as for example Synchronous Optical Network(SONET) or Synchronous Digital Hierarchy (SDH)) links. In particularembodiments, one or more links 120 each include an ad hoc network, anintranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, aportion of the Internet, a portion of the PSTN, a cellulartechnology-based network, a satellite communications technology-basednetwork, another link 120, or a combination of two or more such links120. Links 120 need not necessarily be the same throughout networkenvironment 100. One or more first links 120 may differ in one or morerespects from one or more second links 120. For example, a link 120 toconnected device 130 a may differ from a link 120 to connected device130 b and 130 c. As another example, a link 120 to sensor 160 a maydiffer in one or more respects from a link 120 to sensor 160 b and 160c.

In particular embodiments, an interactive region monitoring system mayinclude one or more connected devices 130, data stores 140, interactiveregion monitoring servers 150, or sensors 160. In particularembodiments, an interactive region monitoring system may be connected toone or more connected devices 130. Connected devices 130 may include,without limitation, a television set, a mobile device such as asmartphone or tablet, a personal computer, a laptop computer, a gamingconsole, a thermostat, a media device, or any other appropriate device.Connected devices 130 may, as an example, communicate with network 110via a wireless communications protocol, such as Wi-Fi or BLUETOOTH. Insome embodiments, interactive region monitoring server 150 may send aninstruction to a connected device, such as an instruction to: turn on,turn off, tune to a particular channel, play audio, play video, or anyother instruction.

In particular embodiments, an interactive region monitoring system maybe connected to one or more sensors 160. Sensors 160 may include anyappropriate sensor, include as an example and not by way of limitation,a depth sensor, an image sensor, a digital camera, a red-green-blue(RGB) camera, an infrared camera, a time-of-flight camera, astructured-light camera, a laser scanner, a radar, a LIDAR, a sonar, aproximity sensor, a motion detector, any other suitable sensor, or anysuitable combination thereof. Sensors 160 may send data to interactiveregion monitoring server 150.

In some embodiments, data store 140 may include networked storage, asillustrated in FIG. 1, such as cloud storage or other network accessiblestorage. Additionally or alternatively, data store 140 may include localstorage within or directly attached to interactive region monitoringserver 150, such as solid state drives (“SSDs”) or hard disk drives(“HDDs”).

In particular embodiments, data store 140 may store a data structurecorresponding to an interactive region. Such a data structure mayinclude, as an example and not be way of limitation, a point cloud, awire-frame model, a parametric-based model, a boundary representation, abinary space partitioning tree, a k-dimensional tree, an octree, aconstructive solid geometry tree, a grid, a polygon mesh model, asurface model, a solid computer-aided design (CAD) model, or any othersuitable data structure.

In particular embodiments, data store 140 may store template datacorresponding to an event or event condition. Such a data structure mayinclude, as an example and not by way of limitation, a template image ortemplate skeletal tracking data. Additionally or alternatively, datastore 140 may store algorithmic data corresponding to an eventcondition. As an example and not by way of limitation, data store 140may store an algorithm that corresponds to an event condition that is aof a user's hand moving to the left at least 13 inches and then movingup at least 5 inches, but only if the time delay between the twomovements is less than 1.3 seconds and only if the rest of the user's isrelatively stable.

In particular embodiments, data store 140 may store data correspondingto an action. In some embodiments, such data may include data about oneor more connected devices 130. As an example and not by way oflimitation, such data may include a unique device identifier,information about the user of a connected device, device details (e.g.,manufacturer, operating system, localized language, audio-visualcapabilities, battery capacity, network connectivity specifications,etc.), details about software applications on a connected device (e.g.,name, version, maximum message size, etc.), or other appropriate devicedata. In some embodiments, data store 140 may store data about anapplication programming interface (API) that corresponds to one or moreconnected devices 130 or one or more software applications configured torun on one or more connected devices 130. In some embodiments, datastore 140 may store data that indicates an association betweeninteractive regions, predetermined events, and predetermined actions.

In particular embodiments, data store 140 stores historical data.Historical data may include (1) one or more interactive regions thathave been created and/or in which user interaction has occurred; (2) atime, date, or duration of user interactions; (3) data from aninteractive region monitoring system; (4) information aboutpredetermined events or event conditions; (5) information aboutpredetermined actions; or (6) any other appropriate data, or anysuitable combination thereof.

In some embodiments, interactive region monitoring server 150communicates with one or more sensors 160. In some embodiments,interactive region monitoring server 150 communicates to one or moresensors 160 instructions to monitor an interactive region. For example,interactive region monitoring server 150 may communicate with data store140 and retrieve a data structure corresponding to an interactiveregion. Based on the data structure, interactive region monitoringserver 150 may direct one or more sensors 160 to monitor thecorresponding interactive region for one or more or event conditions. Insome embodiments, sensors 160 may not monitor portions of a physicalenvironment that do not correspond to an interactive region.

In particular embodiments, an interactive region service 152 ofinteractive region monitoring server 150 may generate an appropriateform for communication with sensors 160 or data store 140. For example,interactive region service 152 may retrieve data from data store 140that corresponds to an interactive region. Interactive region service152 may also retrieve data from one or more sensors 160. Based on acomparison of the data corresponding to an interactive region and thedata from one or more sensors 160, interactive region service 152 mayinstruct one or more sensors 160 to monitor an interactive region.

In particular embodiments, an event detection service 154 of interactiveregion monitoring server 150 may determine whether an event conditionhas occurred based on template data. For example, event detectionservice 154 may retrieve template data corresponding to an eventcondition from data store 140. Template data may include data fromexemplars of an event condition (e.g., using data based on an exemplarprovided by a user). Event detection service 154 may then compare thetemplate data to the data retrieved from one or more sensors 160 todetermine whether the event condition is detected. Additionally oralternatively, event detection service 154 may determine whether anevent condition has occurred based on template data algorithmic datacorresponding to an event condition, as described above. In someembodiments, event detection service 154 may detect a predeterminedevent by determining one or more event conditions associated with thepredetermined event and based on the detection of the one or more eventconditions.

In some embodiments, interactive region monitoring server 150communicates with one or more connected devices 130. In someembodiments, interactive region monitoring server 150 communicates toone or more connected devices 130 to initiate a predetermined action.For example, interactive region monitoring server 150 may communicatewith data store 140 and retrieve data corresponding to a predeterminedaction. Based on the data, interactive region monitoring server 150 maysend an instruction to one or more connected devices 130 to initiate apredetermined action.

In particular embodiments, an action initiation service 156 ofinteractive region monitoring server 150 may initiate a predeterminedaction based on an association between interactive regions, events, andactions. For example, action initiation service 156 may retrieve datafrom data store 140 that indicates an association between interactiveregions, events, and actions. Action initiation service 156 may alsoreceive data from interactive region service 152 and event detectionservice 154 that indicates that an event in an associated interactiveregion has been detected. Based on this, action initiation service 156may use the data indicating an association between interactive regions,predetermined events, and predetermined actions to cross-reference theinteractive region and predetermined event with the associatedpredetermined action. Action initiation service 156 may then initiatethe associated predetermined action.

In particular embodiments, an action initiation service 156 maydetermine an association between interactive regions, events, andactions based on a record for an interactive region. In someembodiments, the interactive region monitoring system maintains a datastructure that includes a number of records (e.g., by storing the recordin data store 140 and updating the record via instructions frominteractive region monitoring server 150). Each record may define one ormore interactive regions for monitoring and may be linked to one or moreusers. For example, each record can include data indicative of thecoordinates of the interactive region, an event that triggers an action,the action that occurs in response to the event, and, in someembodiments, the device that performs the action. The interactive regionmonitoring system may generate a record based on user input (e.g., whena user configures an interactive region, defines an event, defines anaction, etc.).

In particular embodiments, an action initiation service 156 may initiatean action by generating an appropriate instruction to send to one ormore connected devices 130. For example, action initiation service 156may retrieve data from data store 140 that corresponds to apredetermined action. The action may be performed by one or more ofconnected devices 130. Action initiation service may also retrieve fromdata store 140 about the one or more connected devices 130. Based on thedata about the one or more connected devices 130, action initiationservice may initiate an action by determining an appropriate instructionand sending that instruction to a connected device. For example, dataabout a mobile phone may include a phone number. In this example, if anaction is to send a text message to the mobile phone, the actioninitiation service may use the phone number to uniquely identify themobile phone for delivery of the text message (e.g., by emailing a shortmessage service (SMS) gateway).

FIG. 2 illustrates an example embodiment of an interactive region in anexample physical environment. In particular embodiments, depth sensor210 generates a depth image as an output. A depth image may be an arrayof pixels corresponding to regions of a physical environment with adepth value for each pixel indicating the depth of the correspondingregion. A depth image may be generated, as an example and not by way oflimitation, by: stereo triangulation, sheet of light triangulation,structured light, interferometry, coded aperture, or any otherappropriate means of detecting a depth image. In particular embodiments,depth sensor 210 may include or be a part of an electronic system whichincludes an image sensor, a digital camera, a red-green-blue (RGB)camera, an infrared camera, a time-of-flight camera, a structured-lightcamera, a laser scanner, a radar, a LIDAR, a sonar, a proximity sensor,a motion detector, any other suitable sensor, or any suitablecombination thereof. In particular embodiments, depth sensor 210 (or anelectronic system of which depth sensor 210 is a part) may also detectsound, chemical particles, electric current, electric potential,magnetic fields, air flow, light, pressure, force, temperature, heat,motion, images, time, date, biometric data, or any other data. In someembodiments, depth sensor 210 (or an electronic system of which depthsensor 210 is a part) may detect data about the movement of objects orpeople or the speed of objects or people.

In particular embodiments, an interactive region monitoring system maymonitor (e.g., by depth sensor 210) one or more interactive regions,such as interactive region 220, for one or more predetermined events. Aninteractive region may take any suitable size, shape, or location withinphysical environment. As illustrated in FIG. 2, in particularembodiments an interactive region may be defined by a three-dimensionalshape. In particular embodiments, an interactive region may be (or verynearly be) a surface, such as the surface of a table.

In particular embodiments, an interactive region monitoring system maymonitor (e.g., by depth sensor 210) one or more interactive regions,such as interactive region 220, for one or more predetermined events. Insome embodiments, a predetermined event may consist of two or more eventconditions that collectively make up the event, i.e., a predeterminedaction may be initiated only when multiple events (which collectivelyare the predetermined event) are detected. As used herein, apredetermined event may occur within an interactive region when one ormore event conditions making up the predetermined event occur within theinteractive region. For example, a predetermined event may include twoevent conditions: 1) that a particular user enter an interactive regionaround or on a desk chair; and 2) that the temperature in the room bemore than a threshold temperature. In this example, both eventconditions may be required to trigger a predetermined action, such asturning on a fan or air conditioner. This disclosure contemplates that apredetermined event may be equivalent to an event condition or that apredetermined event may require the occurrence of multiple eventconditions. This disclosure contemplates that, where suitable,disclosure relating to event conditions applies to predetermined events,and vice versa.

In particular embodiments, predetermined events and/or event conditionsmay include: movement within an interactive region, the presence orabsence of an object or person within an interactive region, an objector person entering an interactive region, an object or person leaving aninteractive region, or any other suitable event condition. In someembodiments, an event condition may be that other event conditions occura particular order, within a particular time of one another, or anyother relational condition. In particular embodiments, an eventcondition may include output from any sensor or device, such as thesensors or devices illustrated in FIG. 1. In particular embodiments, oneor more event conditions may be determined by a user, e.g., from a listof event conditions. In some embodiments, an interactive regionmonitoring system may detect a predetermined event by receiving datafrom depth sensor 210, a connected device, or by a combination ofdevices that indicate that event conditions have occurred.

In some embodiments, a list of event conditions may include eventconditions that do not require resource-intensive image processing, suchas movement, a level of brightness, an object or user entering orleaving the interactive region, or any other suitable event. Inparticular embodiments, an event condition may include recognizing aparticular object or object type, recognizing a user (such as the user'sface or a client device of the user), recognizing a particular gestureor body position, or any other event condition.

In particular embodiments, a predetermined event or an event conditionmay not be detected unless a threshold event is detected. Detecting athreshold event may include receiving information (e.g., from a depthsensor, a connected device, a database, etc.) that a threshold event hasbeen detected. As an example, an interactive region monitoring systemmay monitor interactive region 220 via depth sensor 210 to detect thepresence of a person as a threshold event. Upon detecting user 230 ininteractive region 220, the interactive region monitoring system maymonitor interactive region 220 for particular user 230, for example, byprocessing the image captured by the depth sensor using facialrecognition, which may be more resource-intensive to accomplish than isdetection and processing of the triggering event. As this particularexample illustrates, a relatively low resource-intensive threshold eventmay be detected and processed before one or more relativelyresource-intensive tasks associated with detecting an event, therebyconserving computer resources during periods where the threshold eventis not detected.

In particular embodiments, one or more connected devices communicativelycoupled to the interactive region monitoring system may be used todetect a predetermined event or an event condition. For example, aparticular user may have a connected device which communicatesinformation that uniquely identifies its user. For example, user 230 mayhave a mobile phone, smart watch, tablet, laptop, or other device thatcommunicates to an interactive region monitoring system (e.g., viaBLUETOOTH) a unique identifier (e.g. a username, a phone number, a mediaaccess control (MAC) address, etc.) that identifies user 230. In anexample embodiment, the interactive region monitoring system maydetermine whether a threshold condition has occurred. In response to adetermination that a threshold condition has occurred, the interactiveregion monitoring system may broadcast a request for proximate connecteddevices to reply with information that uniquely identifies their usersor the respective user devices.

In particular embodiments, an interactive region may be associated withone or more particular users. For example, an interactive region mightbe associated with user 230. In particular embodiments, the associationmay be determined from a data store, such as data store 140, thatcorrelates the region and the user, for example through a profile forthe user, a profile for the region, or both. In particular embodiments,the interactive region may not be monitored for a predetermined eventunless a particular user's presence is detected in the interactiveregion. A user's presence may be detected by, for example, facialrecognition. In another example, as discussed above, interactive regionmay broadcast a request for proximate connected devices to reply withinformation that uniquely identifies the user or users of the connecteddevice. In this example, the interactive region may be monitored for apredetermined event only if a proximate connected device identifies theparticular user associated with the interactive region. Additionally oralternatively, an interactive region may not be associated with anyparticular user.

In particular embodiments, an interactive region monitoring system maymonitor an interactive region only when a threshold event is detected.For example, an interactive region may only be monitored during certaintimes of day, if an on/off toggle is on, or if any other event conditionor predetermined event is satisfied.

In particular embodiments, a predetermined action may be initiated by aninteractive region monitoring system when a predetermined event isdetected, for example when one or more event conditions for thepredetermined event are detected. Event conditions may include detectinga predetermined event within an interactive region and may include oneor more other event conditions, such as the time of day; output from adevice (such as a smartphone, watch, appliance, etc.); an environmentalcondition such as light, noise, temperature, etc.; or any other suitableevent condition. A predetermined action may be performed by aninteractive region monitoring system, by a device connected to aninteractive region monitoring system, by a depth sensor, or by any othersuitable device. As an example, an interactive region monitoring systemincluding depth sensor 210 may monitor interactive region 220 for thepredetermined event of a user entering interactive region 220. Theinteractive region monitoring system may also connect to TV 240. Whenuser 230 enters interactive region 220, the interactive regionmonitoring system may initiate the predetermined action of TV 240turning on by sending an instruction to TV 240. In some embodiments, apredetermined action may include delivering a message or reminder to auser. For example, user 230 may have a history of watching the show‘Game of Thrones.’ If user 230 enters interactive region 220, aninteractive region monitoring system may initiate the playing of anaudio reminder that informs user 230 that the season premiere of ‘Gameof Thrones’ will start in 10 minutes.

In particular embodiments, a predetermined event may include multipleevent conditions. In some embodiments, one event condition may be thatother event conditions occur a particular order, within a particulartime of one another, or any other relational condition. In particularembodiments, an interactive region may be monitored for multiplepredetermined events, each of which triggers a predetermined action. Asan example, an interactive region monitoring system may be monitoringinteractive region 220 via depth sensor 210 for the predetermined eventincluding the event condition of a user entering interactive region 220.When the event condition is satisfied by the detection of a userentering interactive region 220, the interactive region monitoringsystem may initiate the predetermined action of turning on TV 240.Additionally, the interactive region monitoring system may be monitoringinteractive region 220 for an additional predetermined event includingthe event condition that particular user 230 enters interactive region220. If user 230 enters interactive region 220, then the interactiveregion monitoring system may initiate the predetermined action of tuningTV 240 to a particular channel particular to user 230. As anotherexample, an interactive region monitoring system may be monitoringinteractive region 220 for the predetermined event including the eventconditions of a user leaving and TV 240 being on. If the interactiveregion monitoring system detects that TV 240 is on and that a user leftinteractive region 220, the interactive region monitoring system mayinitiate the predetermined action of pausing video content on TV 240.

In particular embodiments, an interactive region monitoring system mayenhance a physical object or surface in an environment. For example, aninteractive region monitoring system may enhance a retail store displayby allowing a user to touch or come near a particular item anddisplaying the price for the item in response. In that example, theinteractive region would include all or some of the item or a space nearthe item, the predetermined event would be contacting or coming near theitem, and the predetermined action would be displaying the price of theitem. As another example, an interactive region monitoring system may beused in a museum to link objects to content on an audio tour. As anotherexample, an interactive region monitoring system may be used in aclassroom to link different parts of the inside of a computer todifferent video tutorials or to the display of more detailed informationabout a particular computer component. As another example, aninteractive region monitoring system may be used at a corporate eventwhen unveiling a new product to trigger appropriate content in apresentation when interaction with a prototype of the product, such astouching the prototype, occurs. As described more fully herein, thisdisclosure contemplates that an interactive region may include anysuitable space or combination of spaces; a predetermined event mayinclude one or more suitable event conditions; and a predeterminedaction corresponding to a predetermined event may include any suitableone or more actions.

FIG. 3A-D illustrates example embodiments of user 230 configuring aninteractive region. In some embodiments, configuring an interactiveregion may include determining an interactive region (e.g., determiningthe shape, area, location, size, orientation, position, or otherproperties of an interactive region), moving an existing interactiveregion, modifying an existing interactive region, or deleting anexisting interactive region. In some embodiments, an interactive regionmonitoring system may store data that represents properties of aninteractive region. For example, an interactive region monitoring systemmay store a data structure corresponding to an interactive region (e.g.,a point cloud, a wire-frame model, a parametric-based model, a boundaryrepresentation, a binary space partitioning tree, a k-dimensional tree,an octree, a constructive solid geometry tree, a grid, a polygon meshmodel, a surface model, a solid computer-aided design (CAD) model, orany other suitable data structure). FIG. 3A illustrates an exampleembodiment where user 230 determines interactive region 320 via agesture. For example, the interactive region monitoring system mayreceive from user 230 user input data that indicates a request to definea new interactive region or to redefine an existing interactive region.In response to receiving the user input data, the interactive regionmonitoring system may receive location data representative of thelocation of the interactive region. The interactive region monitoringsystem may facilitate the user providing location data in a number ofways described below.

In particular embodiments, the interactive region monitoring system maydetect and track, using depth sensor 310, a gesture by user 230 toindicate an interactive region. The shape of the interactive region mayhave a template shape, such as a cuboid (e.g., interactive region 320),a cylinder (e.g., interactive region 350 in FIG. 3B), a polyhedron, apolytope, a sphere, an ellipsoid, a volume bounded by a mesh, a surfacearea, or any other shape. As an example, user 230 may gesture by movinga hand along path 330. The interactive region monitoring system maydetect the gesture via depth sensor 310 and may determine theinteractive region 320 based on path 330.

In some embodiments, an interactive region monitoring system may track agesture via a user device. For example, a user may gesture with a handthat is holding a device, and the device may use an inertial sensor, aGPS sensor, or another suitable sensor to generate and send data to theinteractive region monitoring system or other suitable component todetermine an interactive region.

In particular embodiments, the shape of an interactive region may be apredetermined type of shape with a minimized volume such that the user'sgesture is entirely bounded within the shape. As an example, and not byway of limitation, interactive region 320 may be the cube with thesmallest possible volume such that path 330 is within interactive region320. In particular embodiments, an interactive region may take one of anumber of predetermined shapes. In particular embodiments, aninteractive region may take the shape created by the user's gesture.This disclosure contemplates any suitable method for defining aninteractive region, including by using one or more voice commands, oneor more gestures, one or more instruments (such as, e.g. a stylus), etc.

In particular embodiments, a user may indicate that they desire todefine an interactive region. For example, user 230 may indicate thedesire to define in interactive environment by causing a connecteddevice to send information indicating the desire to an interactiveregion monitoring system. As an example, and not by way of limitation,user 230 may have a software application installed on a connected devicewhich sends data via a network to an interactive region monitoringsystem. After receiving an indication of a user's desire to define aninteractive region, an interactive region monitoring system may monitorthe user's gestures and determine an interactive region based on thosegestures. Additionally or alternatively, an interactive regionmonitoring system may detect a voice command signaling a user's desireto define an interactive region. This application contemplates anysuitable method for a user to signal a desire to define an interactiveregion.

In particular embodiments, an interactive region may intersect physicalobjects (e.g., a stove, walls, floors, furniture, etc.). As an example,interactive region 320 may extend downward into stove 340. In particularembodiments, portions of an interactive region that intersect a physicalobject may not be monitored for a predetermined event or eventcondition.

As an example, if interactive region 320 extends downward into stove340, the portion of interactive region 320 that includes stove 340 maynot be monitored for a predetermined event or event condition.

In particular embodiments, an interactive region, a predetermined event,an action, or any suitable combination thereof may be configureddynamically based on one or more predefined conditions. For example, theshape, area, location, size, orientation, or position of an interactiveregion may be modified based on a predefined condition. Such conditionsmay include a time of day, environmental conditions, user preferences,or any other suitable condition. As an example, an interactive regionaround a lamp may become larger as the area near the lamp becomesdarker. As another example, a predetermined event may include a user'spresence in an interactive region and a temperature in the roomcontaining the interactive region that is above a threshold temperature.The corresponding action may include turning on the fan. The speed ofthe fan may change dynamically based on the temperature in the spacewhen the user is present. In some embodiments, an interactive region maybe configured to come into existence, be deleted, activate, ordeactivate based on conditions. For example, a 24-hour grocery store mayuse an interactive region monitoring system to monitor two doorways. Inthis example, one of the doorways may be shut and locked during certaintimes of the day, and the interactive region corresponding to that entrymay be configured to be deactivated during the times of day when thedoor is locked and to reactivate after that time.

FIG. 3B illustrates an example embodiment of a physical environment inwhich multiple interactive regions are present. In particularembodiments, multiple interactive regions may be configured. Themultiple interactive regions may be distinct from one another, partiallyoverlap, or completely overlap.

In particular embodiments, the image data from overlapping interactiveregions are processed or monitored in a manner that may reduceduplication. As an example, interactive region 320 may be withininteractive region 350. An interactive region monitoring system maymonitor the portions of the physical environment that are within bothinteractive region 320 and interactive region 350 using the same data.For example, depth sensor 310 may capture an image of the region ofoverlap between interactive region 320 and interactive region 350, andthat image may be used to monitor both region 320 and region 350. Insome embodiments, an interactive region monitoring system may be able tomonitor a particular region of a physical environment a certain numberof times per second. If a particular region of the physical environmentis contained within multiple interactive regions, the interactive regionmonitoring system may monitor the portions of each interactive regionthat overlaps with another interactive region the same number of timesper second as it would if the particular region were only within one ofthe interactive regions.

In particular embodiments, a predetermined event may include multipleevent conditions detected in one or more regions. As an example, aninteractive region monitoring system may be monitoring both interactiveregion 320 and interactive region 350 via depth sensor 310. In thisexample, a predetermined event may include event conditions that a pothas been placed on the element in interactive region 320 and that a userhas left interactive region 350. If the interactive region monitoringsystem detects the predetermined event, then the interactive regionmonitoring system may initiate the predetermined action of turning onthe element of stove 340 in interactive region 320.

FIG. 3C and 3D illustrate an example embodiment where user 230determines interactive region 320 using client device 360. In particularembodiments, a user may use a client device to aid in determining aninteractive region. As an example, user 230 may wish to determineinteractive region 350 via gesture. In general, an interactive region isnot visible to a user using only the human eye. In particularembodiments, client device 360 may display part of the physicalenvironment via a camera. Client device 360 may also receive data thatdetermines an interactive region (e.g., the location, position, size,shape, etc.). Client device 360 may display to user 230 an image thatincludes a visual representation of the interactive region in thephysical environment. In particular embodiments, a user may be viewing aclient device as they are gesturing and the client device may display aninteractive region determined by the gesture as the user is gesturing.In particular embodiments, a client device may display only someinteractive regions but not others. For example, FIG. 3D shows onlyinteractive region 350, but not interactive region 320.

In particular embodiments, a user may determine an interactive regionvia a display of a client device. For example, client device 360 mayhave software that allows user 230 to determine interactive region 350.The client device may display an image representative of the physicalenvironment (e.g., via a camera). The user may interact with the clientdevice to determine an interactive region. For example, user 230 may useclient device 360 to determine interactive region 350 by choosing ashape from a list of shapes and using a touch screen to indicate theposition or size of interactive region 350. As another example, user 230may use client device 360 to determine interactive region 350 by drawingthe region on a display of client device 360. This disclosurecontemplates any suitable method of a user interacting with a clientdevice to determine an interactive region, including voice command,interacting with a user interface (UI), gesture, or any other suitablemethod.

FIG. 4 illustrates multiple example interactive regions in an examplephysical environment. In this example, interactive region 420 containswithin it several interactive regions 420 a-e. In particularembodiments, interactive region 420 and interactive regions 420 a-e maybe associated with predetermined events, event conditions, orpredetermined actions. As an example, an interactive region monitoringsystem may monitor via depth sensor 410 interactive region 420 andinteractive regions 420 a-e for the predetermined event including theevent condition of the presence of a user's hand. In this example, oncethe event condition of a user's hand entering interactive region 420 isdetected, the interactive region monitoring system may initiate thepredetermined action of toggling the state of lamp 330 (e.g., turninglamp 330 off or on), which may be connected to the interactive regionmonitoring system. A user's hand entering interactive region 420 mayalso be a threshold event after which the interactive region monitoringsystem may detect a predetermined event including event conditionscorresponding to the movement of the user's hand between regions 420a-e. For example, if a user's hand enters interactive region 420 andinteractive region 420 c, then lamp 330 may turn from the off state tothe on state. If the user then moves the hand down from interactiveregion 420 c to interactive region 420 e, then the interactive regionmonitoring system may initiate the predetermined action of dimming lamp330 (e.g., by varying the voltage lamp 330 uses to control the intensityof light output). In this example, each region 420 a-e may correspond toa particular level of brightness of lamp 330.

FIG. 5 illustrates an example user interface (UI) for associating orconfiguring interactive regions, predetermined events, event conditions,and predetermined actions. A UI may provide a user with the ability tointeract with client system 360 in such a way as to configureinteractive regions; configure predetermined events or event conditions;configure predetermined actions; and associate interactive regions,predetermined events, event conditions, and predetermined actions withone another. Client system 360 may communicate data that corresponds toa user's interactions to the interactive region monitoring system (e.g.,via network 110). Although FIG. 5 illustrates a graphical UI (e.g., auser interface that uses graphical icons or visual indicators displayedon a client system to provide a user with the ability to interact with aclient system), this disclosure contemplates using a voice userinterface (e.g., a user interface that allows a user to interact with aclient system through voice or speech), a gesture interface (e.g., auser interface that provides a user with the ability to interact with aclient system through the motion or state of some part of the user'sbody), or any other suitable type of UI.

In particular embodiments, a UI may include a list of interactiveregions. In particular embodiments, the list dynamically updates as auser configures interactive regions. In some embodiments, theinteractive regions in the list may be given names. For example, list510 includes interactive regions with the names “product display,”“store entrance,” “cash register,” and “employee only doorway.” In thisexample, the physical environment may be in a retail store. Inparticular embodiments, interactive regions may be named in a mannersuggestive of the location of the interactive region in the physicalenvironment. For example, the “product display” interactive region inlist 510 may be located at a product display in the physicalenvironment, the “store entrance” interactive region in list 510 may belocated at the retail store's entrance in the physical environment, andso on. In some embodiments, the UI allows a user to configure aninteractive region by assigning or altering a name for an interactiveregion.

In particular embodiments, a UI may present a list of event conditions.In particular embodiments, a user may configure a predetermined event byselecting one or more event conditions and associate it with one or moreregions and one or more predetermined actions. In particularembodiments, an event condition may have one or more user-configurablefields. For example, FIG. 5 illustrates an example list 520 of eventconditions. Some event conditions in list 520 have user-configurablefields 520 a-b. For example, the event condition “person enters” has twouser-configurable fields (speed and facial recognition). A user mayconfigure via user-configurable field 520 a the event condition to be aperson enters with a certain speed, greater than a certain speed, lessthan a certain speed, etc. The event condition “person enters” also hasa user-configurable field 520 b, which a user may use to configure theevent condition to be a person with a particular recognized face entersthe interactive region. Although FIG. 5 illustrates a particular examplelist 520, this disclosure contemplates any suitable event conditions orpredetermined events with any number of suitable user-configurablefields.

In particular embodiments, a user may define a predetermined event, anevent condition, or an action via an ostensive definition (e.g.,defining by example via performing or demonstrating an exemplar). Forexample, an interactive region monitoring system may be configured torecognize user input indicating a user's desire to ostensively define apredetermined event or an event condition in a given interactive region.If a user indicates such a desire, then the interactive regionmonitoring system may monitor the interactive region. The user may thenprovide an exemplar of the event condition ‘a person enters theinteractive region’ by entering the interactive region. The interactiveregion monitoring system may then recognize the exemplar and definebased on the exemplar the event condition ‘a person enters theinteractive region.’ In some embodiments, once a predetermined event oran event condition is defined ostensively, the event may be availablefor selection via a UI (e.g., an ostensively defined predetermined eventmay appear in list 520 after it is defined).

In particular embodiments, a UI may include a list of predeterminedactions. For example, list 530 includes several example predeterminedactions. Each action may be further configurable by a user (e.g., by auser-configurable field). For example, the predetermined action “play avideo on a display” may allow a user to select a particular displaydevice and a particular video. In general, predetermined actions may beperformed by any connected device, and a UI may allow a user to selectwhich predetermined action is performed by which connected device. Insome embodiments, a UI may include a list of possible connected devices.In particular embodiments, a UI may include a different list for eachconnected device, each list including only those predetermined actionsthat the particular corresponding connected device is capable ofperforming.

In particular embodiments, a user may define a predetermined event or apredetermined action. For example, a user may define a predeterminedevent or a predetermined action via interactions with a connected deviceor by any other suitable means of defining a predetermined event orpredetermined action. In some embodiments, when a user defines apredetermined event or a predetermined action, data that corresponds tothe user-defined predetermined event or predetermined action may bestored in a data store. In some embodiments, once a user defines apredetermined event or a predetermined action, the user-definedpredetermined event or a predetermined action is available in a list,such as list 520 or 530.

In particular embodiments, a UI may allow a user to associate aninteractive region, a predetermined event and a predetermined action. Inthe example above, a user may select the “product display” interactiveregion from list 510. The user may then associate that interactiveregion with the predetermined event that includes the event condition“person enters” from list 520. The user may then associate theinteractive region and the predetermined event with the predeterminedaction “play a video on a display” from list 530. The user may furtherconfigure the predetermined action to play a video demonstrating the useof a product in the display on a display device near the interactiveregion. In this example, if a person enters the “product display”region, then the interactive region monitoring system may cause thevideo to play on the display device.

In particular embodiments, an interactive region may move as an objectin the physical environment moves. For example, list 510 contains theinteractive region “cash register.” This interactive region maycorrespond to an object cash register in the physical environment of aretail store. The interactive region monitoring system may be monitoringthe interactive region for the predetermined event that includes theevent condition “cash drawer opened,” and may initiate the predeterminedaction “use a camera to record and store video” if the predeterminedevent is detected. In this way, the user may record video of the cashregister using a connected camera when the cash drawer of the cashregister is opened. An employee may be cleaning the area around the cashregister and move the cash register. An interactive region monitoringsystem may detect that the cash register has moved (e.g., by using animage histogram, Canny edge detection, or other suitable means) andcause the interactive region “cash register” to move along with the cashregister.

In particular embodiments, particular interactive regions, predeterminedevents, or event conditions may be prioritized. In particularembodiments, an interactive region monitoring system may have limitedprocessing capabilities, memory, or other computer resources. Aninteractive region, predetermined event, or event condition may beprioritized by using relatively more or fewer computer resources for theinteractive region, predetermined event, or event condition compared toother interactive regions, predetermined events, or event conditions.For example, prioritization may involve using a higher framerate tomonitor an interactive region compared to other interactive regions,processing images from an interactive region at a higher rate or soonerthan images from other interactive regions, performing more or lessresource-intensive types of monitoring, or any other suitable method ofprioritization. For example, an interactive region might be monitoredfor an event condition that involves a high rate of speed. In someembodiments, a user may indicate that the event condition involves ahigh rate of speed or an interactive region monitoring system maydetermine that an event condition involves a high rate of speed based onmachine learning (e.g., based on an analysis of historical data) orbased on an association with a particular type of predetermined event orevent condition. In such an example, the interactive region monitoringsystem may prioritize interactive regions where an event conditioninvolving a high rate of speed is anticipated by monitoring theinteractive regions at a higher framerate than is used for otherinteractive regions. As another example, the event condition “cashregister opens” or a predetermined event including that event conditionmay be prioritized based on the importance of the interactive region orthe predetermined event. In some embodiments, there may be a storedvariable indicating the importance of an interactive region orpredetermined event, which may be modified by an interactive regionmonitoring system or by a user.

In particular embodiments, particular interactive regions, predeterminedevents, or event conditions may be automatically prioritized by aninteractive region monitoring system based on a monitoring load. Amonitoring load may be the computer resources (such as the memory,processors, and sensors) required to administer an interactive regionmonitoring system, such as monitoring interactive regions; detectingevents; creating, altering, activating, or deactivating particularinteractive regions, events, and actions; etc. In some embodiments, asinteractive regions, predetermined events, or event conditions aremonitored, the monitoring load may become greater than the computerresources available to an interactive region monitoring system. Inparticular embodiments, an interactive region monitoring system mayprioritize any suitable aspect of the interactive region monitoringsystem (such as regions or portions of regions monitored, eventsmonitored, actions initiated, etc.) based on a monitoring load, such aswhen the monitoring load reaches a threshold monitoring load. Inparticular embodiments, prioritizing reduces the monitoring load.Examples of prioritization may include: using a lower framerate tomonitor an interactive region, delaying the processing of images from aninteractive region, performing less resource-intensive types of resourcemonitoring, or any other suitable method of prioritization. In someembodiments, prioritization based on a monitoring load may by dynamic.For example, an interactive region monitoring system may beautomatically prioritized by an interactive region monitoring systembased on a monitoring load at a certain time interval or in response tocertain triggers (e.g., after an interactive region is configured).

In particular embodiments, a predetermined action may occur at anylocation. In some embodiments, a predetermined action may occur withinan interactive region. For example, the interactive region “employeeonly doorway” in list 510 may contain a door within the interactiveregion. A user may associate the interactive region with thepredetermined event including the event condition “person enters” inlist 520 and use user-configurable field 520 b to indicate that the facemust be recognized as an employee. The user may further associate theinteractive region and predetermined event with the predetermined action“unlock a door” in list 530 and configure that action to unlock the doorwithin the interactive region. In this example, when an employee entersthe interactive region, the interactive region monitoring system maydetect the employee, recognize the employee's face, and then unlock thedoor. Additionally or alternatively, in some embodiments, apredetermined action may occur outside of an interactive region. As anexample, a user may associate the interactive region “store entrance” inlist 510 with the predetermined event including the event conditions“movement” in list 520 and that the retail store is closed (e.g.,detected by determining the time of day compared to the retail storeshours of operation). The user may associate the interactive region andpredetermined event with the predetermined action “send a text messageto a phone number” from list 530 to send a text message to the user'sphone indicating that someone may be in the retail store after it hasclosed. In this example, if the interactive region monitoring systemdetects movement in the “store entrance” interactive region and thestore is closed, it may cause a text message to be sent to the user,where the text message may be sent or received outside of anyinteractive region.

FIG. 6 illustrates an example method 600 implemented by an exampleinteractive monitoring system. In step 610, an interactive regionmonitoring system may receive input identifying an interactive region.In particular embodiments, the input may be from a user. As discussedabove, a user may configure an interactive region by providing inputidentifying an interactive region via gesture, a connected device, orany other suitable means. In some embodiments, a data store of theinteractive region monitoring system may store a data structurecorresponding to an interactive region, as described above in greaterdetail in connection with FIG. 1.

In step 620, the interactive region monitoring system may monitor one ormore interactive regions for a predetermined event, including one ormore event conditions. In particular embodiments, the interactive regionmonitoring system may monitor an interactive region by monitoring thephysical environment that the interactive region is a part of. Inparticular embodiments, the physical environment monitored by theinteractive region monitoring system may include regions outside of theinteractive regions (e.g., an entire field of view of the sensor, room,or building).

In particular embodiments, the physical environment monitored by theinteractive region monitoring system may include only interactiveregions, i.e., the system or a particular system component may monitoronly interactive regions or portions of interactive regions. Inparticular embodiments, only portions of regions monitored by the systemmay be processed by the system, as described more fully herein. Forexample, the interactive region monitoring system may access the datastore 140 to determine the locations of the interactive regions in thephysical space. The interactive region monitoring system may selectivelyprocess portions of the sensor data that correspond to the determinedinteractive regions.

In particular embodiments, the interactive region monitoring system maymonitor for a predetermined event by receiving data from one or moresensors. In some embodiments, as discussed above, data from one or moresensors may be sent to an event detection service of an interactiveregion monitoring server, which may then use the data received to detecta predetermined event.

In step 630, the interactive region monitoring system may detect apredetermined event in an interactive region. As discussed above, apredetermined event may include one or more event conditions, (e.g., achange within the interactive region, receiving information,environmental conditions, etc.). In particular embodiments, an eventdetection service of an interactive region monitoring server maydetermine whether a predetermined event has been detected based on thedata received in step 620.

In particular embodiments, the interactive region monitoring system maydetected a predetermined event based at least in part on a depth imagefor the interactive region (e.g., via a depth sensor). As discussedabove, a depth image may be generated from a depth sensor, as an exampleand not by way of limitation, by: stereo triangulation, sheet of lighttriangulation, structured light, interferometry, coded aperture, or anyother appropriate means of detecting a depth image. Other data may alsobe acquired in step 620, including data related to the detection ofsound, chemical particles, electric current, electric potential,magnetic fields, air flow, light, pressure, force, temperature, heat,motion, images, time, date, biometric data, or any other data. Inparticular embodiments, the depth image may be used, for example, todetermine whether an object has entered, is present in, or has left aninteractive region. In particular embodiments, an event detectionservice of an interactive region monitoring server may retrieve thedepth image from one or more sensors.

In step 640, the interactive region monitoring system may initiate apredetermined action if a predetermined event is detected in step 630.As discussed above, a predetermined action may take place within aninteractive region, outside an interactive region, or both. Apredetermined action may be performed by the interactive regionmonitoring system, by a connected device, or by any other suitablemeans. In some embodiments, as discussed above, a predetermined actionmay be initiated by an action initiation service of an interactiveregion monitoring server.

Particular embodiments may repeat one or more steps of the method ofFIG. 6, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 6 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 6 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method implementedby an example interactive region including the particular steps of themethod of FIG. 6, this disclosure contemplates any suitable methodimplemented by any suitable interactive region including any suitablesteps, which may include all, some, or none of the steps of the methodof FIG. 6, where appropriate. Furthermore, although this disclosuredescribes and illustrates particular components, devices, or systemscarrying out particular steps of the method of FIG. 6, this disclosurecontemplates any suitable combination of any suitable components,devices, or systems carrying out any suitable steps of the method ofFIG. 6.

FIG. 7 illustrates an example method 700 for monitoring for apredetermined event or an event condition and detecting thepredetermined event based on one or more images captured at least inpart by a depth sensor. In particular embodiments, steps 620, 630 ofFIG. 6 may be performed by using method 700. For instance, steps 702,704 of FIG. 7 may correspond to step 620 of FIG. 6 and steps 706, 708 ofFIG. 7 may correspond to step 630 of FIG. 6. Method 700 may begin withstep 702, where a processor of a computing device, such as any suitablecomputing device described herein, may receive data representing animage of at least a portion of an interactive region.

At step 704, a portion of the image may be characterized as backgroundor other non-event information. For example, the method may refer toprevious images of the environment and identify portions that have notchanged, or have not changed more than a particular threshold, asbackground. In particular embodiments, only background or non-event datafor the portion of the image corresponding to the interactive region maybe identified. In particular embodiments, the background of an image maybe subtracted from the image prior to processing the image. Inparticular embodiments, background subtraction may be accomplished byframe differencing, using a mean filter, running a Gaussian average,using a background mixture model, or any other suitable method. Inparticular embodiments, background subtraction may isolate objects inthe foreground of an image to aid in the detection of movement, objectdetection, object localization, object recognition, object tracking,semantic segmentation, or other forms image processing. In particularembodiments, background subtraction may allow data that indicates change(e.g., movement, change in color, etc.) to be isolated from other data.In particular embodiments, only background corresponding to theinteraction region may be subtracted.

At step 706, a portion of the image not characterized as background ornon-event data may be processed to determine whether the portioncontains information indicating that an event or event condition hasoccurred. Image processing may include measuring the level of thebackground-subtracted information and comparing the level to athreshold. If the level is greater than the threshold, then it may bedetermined that an event or event condition has occurred. Additionallyor alternatively, image processing may also include object recognition,facial recognition, or any other type of image processing. Any suitabletechniques may be used to process an image, for example and not by wayof limitation: edge matching, detection of a change in lighting orcolor, greyscale matching, using a histogram, comparing an image to aone or more other images, using an interpretation tree, etc. Inparticular embodiments, data identifying or an event or event conditionmay be accessed from a profile or record particular to the interactiveregion being analyzed, as described more fully above. In particularembodiments, step 706 may include skeleton tracking (e.g., usingskeleton data to infer a body position of a person or a gesture made bya person). In particular embodiments, skeleton tracking may beaccomplished by using a model representing a human target, where themodel may include adjustable skeletal points corresponding to the jointson a human. In some embodiments, the model may be adjusted intodifferent poses corresponding to poses performed by the human target. Insome embodiments, skeleton data may be determined based on a comparisonof data corresponding to known gestures or poses. In some embodiments,machine learning may be used to determine skeleton data. In particularembodiments, skeleton data may indicate that a person or a part of aperson is within, has entered, or has left a particular interactiveregion.

If no event or event condition is detected in step 706, method 700 mayend, and step 620 of FIG. 6 may be performed. If an event is detected,method 700 may end, and step 640 of FIG. 6 may be performed. Inparticular embodiments, method 700 may include step 708, where themethod determines whether a predetermined event requires any additionalevent condition(s), and if so, whether those event condition(s) havebeen satisfied. As described more fully herein, the occurrence of otherevent conditions may be determined by processing the image captured bythe depth sensor or determined from output from any other suitablesensor. If other event conditions are required to detect an event and atleast one of those event conditions is not satisfied, then the methodmay end, and step 620 of FIG. 6 may be performed. If an event does notinclude additional event conditions, or if all the required additionalevent conditions are satisfied, then the method may end, and step 640 ofFIG. 6 may be performed.

Particular embodiments may repeat one or more steps of the method ofFIG. 7, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 7 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 7 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method fordetecting a predetermined event, or an event condition, including theparticular steps of the method of FIG. 7, this disclosure contemplatesany suitable method for detecting a predetermined event, or an eventcondition, which may include all, some, or none of the steps of themethod of FIG. 7, where appropriate. Furthermore, although thisdisclosure describes and illustrates particular components, devices, orsystems carrying out particular steps of the method of FIG. 7, thisdisclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 7.

FIG. 8 illustrates an example computer system 800. In particularembodiments, one or more computer systems 800 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 800 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 800 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 800.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems800. This disclosure contemplates computer system 800 taking anysuitable physical form. As example and not by way of limitation,computer system 800 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a laptop or notebook computer system, an interactive kiosk, amainframe, a mesh of computer systems, a mobile telephone, a personaldigital assistant (PDA), a server, a tablet computer system, anaugmented/virtual reality device, or a combination of two or more ofthese. Where appropriate, computer system 800 may include one or morecomputer systems 800; be unitary or distributed; span multiplelocations; span multiple machines; span multiple data centers; or residein a cloud, which may include one or more cloud components in one ormore networks. Where appropriate, one or more computer systems 800 mayperform without substantial spatial or temporal limitation one or moresteps of one or more methods described or illustrated herein. As anexample and not by way of limitation, one or more computer systems 800may perform in real time or in batch mode one or more steps of one ormore methods described or illustrated herein. One or more computersystems 800 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate. [83] In particular embodiments, computer system 800includes a processor 802, memory 804, storage 806, an input/output (I/O)interface 808, a communication interface 810, and a bus 812. Althoughthis disclosure describes and illustrates a particular computer systemhaving a particular number of particular components in a particulararrangement, this disclosure contemplates any suitable computer systemhaving any suitable number of any suitable components in any suitablearrangement.

In particular embodiments, processor 802 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 802 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 804, or storage 806; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 804, or storage 806. In particular embodiments, processor802 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 802 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 802 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 804 or storage 806, andthe instruction caches may speed up retrieval of those instructions byprocessor 802. Data in the data caches may be copies of data in memory804 or storage 806 for instructions executing at processor 802 tooperate on; the results of previous instructions executed at processor802 for access by subsequent instructions executing at processor 802 orfor writing to memory 804 or storage 806; or other suitable data. Thedata caches may speed up read or write operations by processor 802. TheTLBs may speed up virtual-address translation for processor 802. Inparticular embodiments, processor 802 may include one or more internalregisters for data, instructions, or addresses. This disclosurecontemplates processor 802 including any suitable number of any suitableinternal registers, where appropriate. Where appropriate, processor 802may include one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 802. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 804 includes main memory for storinginstructions for processor 802 to execute or data for processor 802 tooperate on. As an example and not by way of limitation, computer system800 may load instructions from storage 806 or another source (such as,for example, another computer system 800) to memory 804. Processor 802may then load the instructions from memory 804 to an internal registeror internal cache. To execute the instructions, processor 802 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 802 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor802 may then write one or more of those results to memory 804. Inparticular embodiments, processor 802 executes only instructions in oneor more internal registers or internal caches or in memory 804 (asopposed to storage 806 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 804 (as opposedto storage 806 or elsewhere). One or more memory buses (which may eachinclude an address bus and a data bus) may couple processor 802 tomemory 804. Bus 812 may include one or more memory buses, as describedbelow. In particular embodiments, one or more memory management units(MMUs) reside between processor 802 and memory 804 and facilitateaccesses to memory 804 requested by processor 802. In particularembodiments, memory 804 includes random access memory (RAM). This RAMmay be volatile memory, where appropriate Where appropriate, this RAMmay be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 804 may include one ormore memories 804, where appropriate. Although this disclosure describesand illustrates particular memory, this disclosure contemplates anysuitable memory.

In particular embodiments, storage 806 includes mass storage for data orinstructions. As an example and not by way of limitation, storage 806may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage806 may include removable or non-removable (or fixed) media, whereappropriate. Storage 806 may be internal or external to computer system800, where appropriate. In particular embodiments, storage 806 isnon-volatile, solid-state memory. In particular embodiments, storage 806includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 806 taking any suitable physicalform. Storage 806 may include one or more storage control unitsfacilitating communication between processor 802 and storage 806, whereappropriate. Where appropriate, storage 806 may include one or morestorages 806. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 808 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 800 and one or more I/O devices. Computer system800 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 800. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 808 for them. Where appropriate, I/O interface 808 mayinclude one or more device or software drivers enabling processor 802 todrive one or more of these I/O devices. I/O interface 808 may includeone or more I/O interfaces 808, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 810 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 800 and one or more other computer systems 800 or one ormore networks. As an example and not by way of limitation, communicationinterface 810 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 810 for it. As an example and not by way of limitation,computer system 800 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 800 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 800 may include any suitable communication interface 810 for anyof these networks, where appropriate. Communication interface 810 mayinclude one or more communication interfaces 810, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

In particular embodiments, bus 812 includes hardware, software, or bothcoupling components of computer system 800 to each other. As an exampleand not by way of limitation, bus 812 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 812may include one or more buses 812, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

Herein, “automatically” and its derivatives means “without humanintervention,” unless expressly indicated otherwise or indicatedotherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. An apparatus comprising: a depth sensor; one ormore computer-readable non-transitory storage media embodyinginstructions; and one or more processors configured to execute theinstructions to: monitor, at least in part based on output of the depthsensor, a physical environment comprising a user-defined interactiveregion for a predetermined event, wherein the user-defined interactiveregion is configured at least in part based on user input; detect, atleast in part based on the output of the depth sensor, the predeterminedevent in the interactive region; and in response to detection of thepredetermined event, initiate a predetermined action associated with thepredetermined event.
 2. The apparatus of claim 1, wherein thepredetermined event includes a touch by the user with respect to anobject in the interactive region.
 3. The apparatus of claim 2, whereinthe predetermined action associated with the predetermined eventincludes displaying information associated with the object.
 4. Theapparatus of claim 3, wherein: the object includes a product; and thepredetermined action associated with the predetermined event includesdisplaying a price associated with the product.
 5. The apparatus ofclaim 3, wherein the predetermined action associated with thepredetermined event includes displaying a video associated with theobject.
 6. The apparatus of claim 3, wherein the information associatedwith the object is displayed by a display device near the interactiveregion.
 7. The apparatus of claim 1, wherein the user input is providedby at least one of: a voice command, a gesture, or an instrument.
 8. Amethod comprising: monitoring, at least in part based on output from adepth sensor, a physical environment comprising a user-definedinteractive region for a predetermined event, wherein the user-definedinteractive region is configured at least in part based on user input;detecting, at least in part based on the output of the depth sensor, thepredetermined event in the interactive region; and in response todetecting the predetermined event, initiating a predetermined actionassociated with the predetermined event.
 9. The method of claim 8,wherein the predetermined event includes a touch by the user withrespect to an object in the interactive region.
 10. The method of claim9, wherein the predetermined action associated with the predeterminedevent includes displaying information associated with the object. 11.The method of claim 10, wherein: the object includes a product; and thepredetermined action associated with the predetermined event includesdisplaying a price associated with the product.
 12. The method of claim10, wherein the predetermined action associated with the predeterminedevent includes displaying a video associated with the object.
 13. Themethod of claim 10, wherein the information associated with the objectis displayed by a display device near the interactive region.
 14. Themethod of claim 8, wherein the user input is provided by at least oneof: a voice command, a gesture, or an instrument.
 15. One or morecomputer-readable non-transitory storage media embodying software thatis operable when executed to: monitor, at least in part based on outputof the depth sensor, a physical environment comprising a user-definedinteractive region for a predetermined event, wherein the user-definedinteractive region is configured at least in part based on user input;detect, at least in part based on the output of the depth sensor, thepredetermined event in the interactive region; and in response todetection of the predetermined event, initiate a predetermined actionassociated with the predetermined event.
 16. The media of claim 15,wherein the predetermined event includes a touch by the user withrespect to an object in the interactive region.
 17. The media of claim16, wherein the predetermined action associated with the predeterminedevent includes displaying information associated with the object. 18.The media of claim 17, wherein: the object includes a product; and thepredetermined action associated with the predetermined event includesdisplaying a price associated with the product.
 19. The media of claim17, wherein the predetermined action associated with the predeterminedevent includes displaying a video associated with the object.
 20. Themedia of claim 17, wherein the information associated with the object isdisplayed by a display device near the interactive region.